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ABSTRACT 



The National Science Foundation's Statewide Systemic 
Initiative (SSI) program attempted to address concerns about student 
performance in science, mathematics, and technology education. The SSIs were 
supposed to increase and improve student learning in these areas by having 
challenging academic standards, a hands-on approach to instruction, the use 
of curricula relevant to students' lives, emphasis on higher-order thinking 
skills, the use of technology in the classroom, and assessments that 
reinforce these instructional practices. Over the last 4.5 years, the New 
Jersey SSI (NJSSI) spent $15 million SSI dollars. This evaluation centered on 
three questions: (1) do students learn more because of the SSI; (2) are they 

better equipped to apply what they are learning to everyday problems; and (3) 
are the inequities in performance among different groups of students being 
reduced? The NJSSI had some reach in 446 schools (19% of the schools in New 
Jersey), a little implementation in 409 schools, and no reach at all in 1,402 
schools. Evaluation data came primarily from state-mandated assessments. No 
evidence was found for the first evaluation question; students in NJSSI 
schools did not learn more than their counterparts in schools without NJSSI 
affiliation. Evidence with respect to the second question was too limited, 
and the question could not be answered. With regard to question 3, it was 
found that inequities in performance had actually worsened. The knowledge gap 
between underserved school districts and the rest of the state increased on 
the state -mandated eighth grade assessment in mathematics during the time the 
NJSSI was in operation. In addition, the knowledge gap between students from 
affluent suburbs and urban students increased in Advanced Placement Calculus, 
Advanced Placement Biology, and Advanced Placement Computer Science. 

(Contains five figures, eight tables, and nine endnotes.) (SLD) 
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Abstract 



Launched 33 years after Sputnik, the National Science Foundation's (NSF) SSI 

program attempted to address concerns about student performance in science, math 

and technology education in the United States. The SSIs were supposed to increase 
and improve student learning of math, science and technology. The SSIs were going 
to accomplish this task by; having challenging academic standards, a hands-on 
approach to instruction, the use of curricula relevant to students' lives, emphasis on 
higher order thinking skills, the use of technology in the classroom, and assessments 
that reinforce the above mentioned instructional practices. 

Over the last 4 1/2 years, the New Jersey Statewide Systemic Initiative (NJSSI) spent 
$15 million SSI dollars; attempting to improve science, math and technology 
education in New Jersey. Policy makers at the state and national level might like to 
know what impact NJSSI had, if any, on improving educational outcomes. Three 
evaluation questions highlighted our inquiry. (1) Are students learning more 
because of the SSI? (2) Are students better equipped to apply what they are leaning to 
everyday problems in the community and in the workplace? (3) Are the inequities in 
performance among different groups of students being reduced? 

We found no evidence for the first evaluation question. Students in schools affiliated 
with NJSSI did not learn more than their counterparts enrolled in schools with no 
NJSSI affiliation. NJSSI did not add value to a school's educational achievement. The 

evidence with respect to evaluation question number 2 was too limited, and we could 

not answer it. With respect to evaluation question number 3, we found that inequities 
in performance got worse. The knowledge gap between the underserved school 
districts and the rest of the state increased on the state mandated 8th grade 
assessment in mathematics during the time NJSSI was in operation. Additionally, the 
knowledge gap between students from affluent suburbs and urban students increased 
in Advanced Placement Calculus, Advanced Placement Biology, and Advanced 
Placement Computer Science. In summary, the equity situation in New Jersey is 
worse today than when the SSI was birthed. 
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Introduction 



The NSF SSI Program 



Over the last eight years, the National Science Foundation (NSF) spent $265 million 
dollars attempting to improve math, science and technology education through its 
Statewide Systemic Initiatives (SSI) Program. Launched 33 years after Sputnik, the 
SSI program attempted to address concerns about student performance in science, 
math and technology education in the United States (CPRE 1995). Additionally, the SSI 
program was supposed to deal with equity issues associated with differential groups 
of student performance; race, ethnicity, and gender. NSF grants, awarded to states, 
were typically $10 million dollars over five years, with required matching funds 
from the state. 

At its peak, 25 states and Puerto Rico participated in the SSI program. In 1996, the 
initial 10 states selected to participate in the program (called Cohort I) completed the 
five year grant. In 1997, an additional 10 states and Puerto Rico (called Cohort II) 
completed the five year grant. In 1998, the last five states (called Cohort III) 
completed their five year grant. 

During the fifth year of a grant, states in each cohort could have competed against 
each other for a renewal grant from the NSF. Renewal grants could have requested 
funding up to $1.4 million dollars a year, but totaling no more than $6 million dollars 
over a five year period. In the first cohort, Connecticut and Louisiana were renewed. 
In the second cohort, Massachusetts, Puerto Rico, Texas and Vermont were renewed. 
The winners of the renewal competition for the third cohort were not known at the 
time this paper was written. 

As the last cohort of states completes their five year grant, the NSF is downsizing the 
SSI program in two different ways simultaneously. The number of states 
participating the SSI program has been reduced. Additionally, the amount of award 
given to the states successful at winning renewal grants was reduced an average of 

40%. At its peak in federal fiscal years 93 and 94, the NSF was spending $53-$54 
million dollars on the SSI program. By federal fiscal year 97, it is anticipated that the 
NSF will spend $18 million dollars on the SSI program (1998 Program Effectiveness 
Reviews (PER) Report Statewide Systemic Initiatives [SSI] Program, 1998). 

With the final states wrapping up their fifth year in the program, it is reasonable to 
ask whether the SSIs had the desired positive impacts on student performance in 
mathematics, science and technology. CPRE (1995) raises three evaluation questions 
pertinent to the SSIs 

1. Are students learning more because of these SSIs? 

2. Are students better equipped to apply' what they are leaning to everyday 
problems in the community and in the workplace? 

3. Are the inequities in performance among different groups of students being 
reduced? 

We attempt to answer these evaluation questions in the context of a case study of one 
SSI, the New Jersey Statewide Systemic Initiative (NJSSI). 
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The SSIs were supposed to increase and improve student learning of math, science 
and technology by; having challenging academic standards, a hands-on approach to 
instruction, the use of curricula relevant to students' lives, emphasis on higher order 
thinking skills, use of technology in the classroom, and assessments that reinforce 
the above mentioned instructional practices. 

As the name implies, the original intention of the Statewide Systemic Initiatives was 
systemic reform. At the start of the program (in 1990) the systemic reform fad was 
strong and gaining strength. Consistent with the systemic reform movement, the 
SSIs were supposed to form broad affiliations with a wide variety of different actors; 
school districts, institutions of higher education, informal education, and business 
and industry, to mention some of the most pertinent participants. Through these 

affiliates, the SSI was supposed to positively impact math, science and technology 
education holistically in a state. The NSF was were not looking for piecemeal change 
from the SSIs. The SSIs were supposed to facilitate a wholesale improvement of SMT 
education in a state. NSF expected that professional development of teachers would be 
a key component in each state's strategy for SMT educational improvement. 

By 1995, the luster had come off the systemic reform fad. In the early years, 
proponents of systemic reform thought that low expectations from the educational 
system (teachers, principals and students) could explain less than satisfactory 
student performance. They reasoned that increasing expectations, and having high 
expectations for all students, would lead to increased student achievement. Evidence 
showed that the patterns of student achievement were unchanged in states that 

adopted the systemic reform model (Fenster, Stufflebeam, Wiersma, Nitko 1995). 
Additionally, despite many attempts to alter the educational system, the system itself 
did not change (Cohen 1995). 

NSF responded to the challenges and limitations demonstrated at the end of the 
systemic reform period by embracing the standards-based curriculum reform 
initiative in the middle 1990's. The NSF did not change the name of the SSIs, but 
changed the focus of the work from systemic reform to standards-based curriculum. 
SSIs were expected to embrace the standards-based curriculum fad, and alter their 
professional development programs to include instruction in standards-base 
curriculum. Virtually all SSIs made this transition during the middle 1990's. By 1997, 
the SSIs were no longer considered systemic reform initiatives. The SSIs were seen 

as standards-based initiatives 1 . However, the shift to a standard-based initiative was 
not a 100% shift in focus. The SSIs were still working towards: a hands-on approach 
to instruction, the use of curricula relevant to students' lives, and an emphasis on 
higher order thinking skills. Additionally, assessment issues and equity concerns 
remained major foci for the SSIs. 

The State of Education In New Jersey 

The most urban state in the United States, New Jersey offers the educational 
researcher a blistering diversity of school districts within close proximity to each 
other. Education in New Jersey is characterized by a long record of local control, and 
a mosaic of many different school districts, which often have overlapping local 
jurisdictions subject to different of Boards of Education. Decentralization continues. 

In the last year alone, an additional 22 functional school districts were created. In the 
1997-1998 academic year, New Jersey has 616 functional school districts. In many 
cases, school districts cover a limited number of grades; k-6, k-8, and 9-12, to mention 
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only a few combinations. Families that are residential stable during the time it takes a 
child to go from k-12 could end up sending that child to two or three different school 
districts. 

Importantly, the extreme decentralization of school districts creates many 
homogenous school districts on racial and economic grounds. The recent "Quality 
Counts" study stated that New Jersey had some of the best (suburban) schools and 
some of the worst (urban) schools in the country (Education Week on the Web-Quality 
Counts '98: The Urban Challenge, 1998). School districts with considerable poverty 
exist in communities of affluence. Around Rutgers University, the City of New 
Brunswick is a "special needs" district, one of the poorest 28 communities in the state. 
All of the school districts surrounding New Brunswick are relatively upscale, 
wealthy suburban enclaves. Some of these communities (South Brunswick, North 
Brunswick, and Highland Park) are quite a bit upscale. The City of New Brunswick is 
a "pocket of poverty in an area of affluence". Such pockets of poverty are typical in 
New Jersey. 

The New Jersey Department of Education 

The New Jersey Department of Education (NJDOE) has some responsibility over every 
school district in the state. The NJDOE requires all 8th grade and 11th grade students 
to take state mandated assessments. In 1997, the NJDOE piloted a 4th grade assessment. 
The 4th grade assessment will be piloted a second time in 1998, before going "live" in 
1999. By introducing a 4th grade assessment, the NJDOE will eventually bring nearly 
all of New Jersey's 2257 schools into the state's assessment program. In the not too 
distant future, nearly every school in the state will be subject to some level of public 
accounting with respect to the academic achievement of their students on state 
mandated assessments. Unfortunately, the elementary schools are currently not 
participating in the state's assessment program, and there is no public accounting of 
student performance in these 1281 schools. 

New Jersey averages spending about $8300 per child for k-12 education, the highest 
per capita expenditure in the United States. However, that $8300 average figure 

varies by school district. In urban districts, far less is spent to educate each child. A 
27 year legal battle, decided in 1997, found that the 28 so called "special needs" 
districts were not receiving a sufficient amount of state aid to educate their students. 
The court initially ordered that 26 out of the 28 "special need districts" receive an 

additional $246 million dollars^. This money was distributed to the 26 districts, 
starting in the 1997-98 academic year. The court has since ruled that an additional 
$312 million dollars needs to be distributed to give school children in these districts 
an appropriate education. 

Three key points for the reader to understand the state of education in New Jersey: 

extreme decentralization, financial inequity, and racial and economic segregation of 

school districts. New Jersey is not different from many states in the northeast and 
midwest with respect to financial inequity and racial and economic segregation. 

However, due to the number of school districts, and the homogeneity of those school 
districts, New Jersey exhibits a far greater degree of economic segregation and 
financial inequity than most states. 
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The New Jersey Statewide Systemic Initiative 



The New Jersey Statewide Systemic Initiative (NJSSI) was birthed in 1993, on the 
Busch Campus of Rutgers University, located in Piscataway (a relatively upscale 
suburban community) one mile north of the City of New Brunswick. 

Faced with a mandate from the NSF to work with urban schools, but recognizing the 
dominant political ethos of New Jersey is suburban, the NJSSI launched two 
programs: 

a professional development initiative, called Thrust I, 
a systemic reform initiative, called Thrust II. 

Much later, in response to a mandate from the NSF to scale-up, NJSSI launched a 
standards-based initiative, called Thrust III, in the summer/fall of 1997. 

After spending nearly $15 million dollars ($10 million from NSF and a $5 million 
matching grant from the NJDOE), over the last 4 1/2 years, policy makers at the state 
and national level might like to know what impact the NJSSI had, if any, on 
improving SMT education in New Jersey. Motivated by the CPRE (1995) questions, we 
conducted an evaluation of the NJSSI to answer the question of impact. 

Accomplishments of the NJSSI 

Reach of NJSSI Professional Development at the School Level 

At the school level, the SSI had some reach in 446 schools (19% of the total number of 
schools in the state), had a little reach in an additional 409 schools (18% of the total), 
and no reach in 1402 schools (62% of the total). Figure 1 presents this information in 
percentages. 

Across the grade levels, NJSSI reached out to schools unevenly. NJSSI concentrated its 
professional development efforts at the lower grade levels (e.g., the elementary 
schools). At the junior high school level, NJSSI had trained at least one teacher in 
281 out of New Jersey's 593 junior high schools, and had no reach in the other 312 
schools. At the high school level, NJSSI had at least some reach in 35 out of New 
Jersey's 383 high schools, and had no reach in the other 348 high schools. 

Student Reach 



Estimating the number of students impact by NJSSI is not easy. For example, consider 
the high schools. There was no NJSSI activity in 348 high schools, enrolling 285,556 
students. The students in these 348 high schools were not impacted by the SSI. 
However, what can we say about students enrolled in the 35 high schools where NJSSI 
was (at least) somewhat active? We know the SSI did not impact every student in 
these 35 schools. Arbitrarily, we assumed the SSI impacted 20% of the students in 
these schools. With that assumption, we concluded that NJSSI reached 2.2% of total 
number of high school students in New Jersey (7,328 out of 322,199). 

Following the logic of the high school, we estimated the number of students reached 
by NJSSI in the middle schools. There was no SSI activity in 312 middle schools, with 
a total enrollment of 151,086 students. The remaining schools, with 184,253 students 
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had some SSI reach. Using the 20% rule, we estimated that the SSI reached 36,850 
students, 11.0% of the 335,339 students enrolled in middle schools in the state of New 
Jersey. 

Combining middle and high school students, we estimated that NJSSI impacted 6.7% of 
the 657,538 students enrolled in the middle and high school grades. 



Figure 1: Percentage of Schools Reached by NJSSI 




Teacher Reach 

At the teacher level, the SSI reached 2600 science, math and technology (SMT) 
teachers intensively, through a "120 hour” professional development program. The 

SSI reached an additional 6000 SMT teachers superficially, generally through one day 
workshops. The 8600 teachers impacted in some way (intensive + superficial) 
represented 20% of the total SMT teachers in New Jersey. Figure 2 presents the 
percentages^. 

In summary, the SSI reached about 20% of the total SMT teachers in New Jersey and 
impacted 7% of the students in schools subject to New Jersey's state mandated 
assessment program. 
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Figure 2: Percentage of Teachers Reached by NJSSI 
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NJSSI’s Impact on Student Achievement 

Limitations of Using State Mandated Assessments to Impact of an Educational Reform 
Initiative. 



There are a number of excellent reasons to avoid using state mandated assessments to 
determine the impact of an educational reform initiative impact. As Haertel (1994) 

noted, with any recurrent assessment, a tradition of past examinations develops, and 
over time, examiners become reluctant to make significant changes from year to 

year because teachers will not know what to teach. If that happens, the domain of 

assessment tasks will grow too narrow. Assessment scores will rise, but instruction 

will become stereotyped. As a consequence, students who have learned to do well on 
particular kinds of items included in the assessment may do poorly on equally valid 
items that are assessing the same skills in a slightly different way. With respect to 
the high stakes, 11th grade assessment that students are required to pass in order to 
graduate in New Jersey, Madaus (1988) noted that high stakes test scores may become 
the most important goal of education. Haertel (1994) pointed out an important 
component of education is lost when teachers and students work for higher scores on 
a high stakes assessment, instead of the intellectual attainments an increasing score 
is supposed to represent. 
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Stecher and Koretz (1996) highlight the corruptibility of repeated administrations of 
state mandated assessments. Scores on state mandated assessments tend to rise faster 
than other assessments, such as the National Assessment of Educational Progress 
(NAEP). The increasing scores on state mandated assessments gives comfort to state 
and local officials (governors, member of the state legislature, workers at the state 
board of education, school boards, superintendents, principals, leaders in business 
and industry, teachers, parents and students) that educational achievement is 
improving at the state and local level. People interested in educational policy at the 
federal level see very small increases in NAEP scores as evidence that educational 
achievement at the state and local level is not improving nearly as robustly as scores 
on state mandated assessments would suggest. The recent TIMSS report showing that 
12th grade students in the United States scored below the international average in 
science and math, and exceeded the performance of only two nations would bolster 
this position (http://www.ed.gov/TIMSS/overview.html). The NSF is very much 
aware of these issues, and has invested time, effort and money to develop 
performance indicators in math and science that avoid using state mandated 
assessments (Stecher and Koretz, 1996). 

A different criticism of using New Jersey's state mandated assessments to evaluate 
NJSSI's achievements comes from Leo Klagholz, State Education Commissioner in New 
Jersey. According to Klagholz, state tests are too easy! "There are districts that can't 
improve anymore." "Their scores have started to level off" (Chiles, 1998). Klagholz 
was referring to scores on the high stakes 11th grade assessment, where some 
schools are so close to a passing rate of 100%, they have no room to improve. 

Klagholz could also have been referring to some middle schools, who have over 95% 
of their 8th grade students in the top proficiency category. 

However, there are some good reasons to use state mandated assessment scores to 

monitor the impact of an educational reform initiative. Schools care about the 
results of state mandated assessments. Results are printed in the newspaper, and 
schools can look good or bad to the outside community as a function of performance 
on an assessment. A favorite assessment of federal policy makers, NEAP, has no 
meaning to local school districts. No school district, school, or student receives a 
score on NEAP. Students, schools and school districts all receive scores on state 
mandated assessments. For this reason, state mandated assessments are far more 
meaningful for students, schools and districts than NEAP. 

We understand the corruptibility of state mandated assessments. New Jersey's state 

mandated assessments have not been substantially changed in over 5 years. Despite 
the corruptibility problem, and the ceiling issue noted by the Commissioner of 
Education, we think that state mandated assessments provide o_n_e piece of evidence 
to assess the programmatic impact of an SSI. Thus, the strategy used in this paper 
follows most closely Upton and Supovitz's (1996) work in estimating the 
programmatic impact of Ohio's SSI. 



System Level Effects: Knowledge Gap Increased on the Eighth Grade State Mandated 

Assessment in Mathematics 

New Jersey assess nearly all 8th grade students, but only reports results on "regular" 
students. Students classified as special education (10% of all students) and limited 
English proficient (2% of all students) are not included in the New Jersey's reporting 
on state mandated assessments. The reporting function thus eliminates 12% of New 
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Jersey's school children. Therefore, analysis of New Jersey's 8th grade assessment is 
limited to 88% of the state's school children, so-called "regular" students. 

The underserved districts compared to the rest of New Jersey 

The underserved population of school children in New Jersey are heavily 
concentrated in so called "special needs districts"--the poorest 28 communities in New 
Jersey. These 28 school districts constitute less than 5% of the total number of school 
districts in the state. However, these districts tend to have large numbers of students, 
and serve over 21% of the school children in New Jersey. Newark, Jersey City and 
Paterson, the three largest school districts in the state, are classified as "special 

needs "4. The vast majority of students in these districts come from families that have a 
lot of financial need. As an example, 85% of the children attending school in the 
Paterson district qualify for free or reduced lunch. 



Figure 3: Increasing Knowledge Gap Between Special Needs 

Districts and the rest of the state of New Jersey in 8th grade 

mathematics, 1994-1997. 




YEAR 



Mathematics . When NJSSI was birthed, the knowledge gap between the 
special need districts and the rest of the state on mathematics was measured at 
34.4 points on the state mandated 8th grade assessment. Over the next four 
years, there was been a slight upward tend in the mean score on the 
mathematics section of the 8th grade assessment for these special needs 
students. In 1994, the mean score for students in special needs districts was 
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102.2, in 1995 104.6, in 1996 106.8., and in 1997 111.5. Over the same period, 
there has been a larger upward trend in the mean score on the mathematics 
section of the 8th grade assessment for students enrolled in all other districts. 

In 1994, the mean score for non-special needs districts was 136.6, in 1995 143.8, 
in 1996 144.3, and in 1997, 148.9. During the years the SSI has been in 
operation, the knowledge gap between the special needs districts and the rest 
of the state INCREASED by 3.0 points on the mathematics component of the 
state-mandated 8th grade assessment. The knowledge gap between the special 
needs districts and the rest of the state with respect to the mathematics 8th 

grade assessment is now 37.4 points. Figure 3 highlights the trend. 

Reading . When NJSSI was birthed, the knowledge gap between the special 

needs districts and the rest of the state on reading was measured at 36.4 points on the 

state mandated assessment. Over the next four years, there was a upward tend in the 
mean score on the reading section of the 8th grade assessment. In 1994, the mean 

score for students in special needs districts was 113.6 in 1994, 116.7 in 1995, 120.0. in 

1996, and 119.1 in 1997 Over the same period, there was a more gradual upward trend 
in the mean score on the reading section on the 8th grade assessment for students 
enrolled in all other districts. The mean score for non-special needs districts was: 

150.0 in 1994; 151.3 in 1995; 152.2 in 1996; and 152.5 in 1997. During the years the SSI 

has been in operation, the knowledge gap between the special needs districts and the 

rest of the state DECREASED by 3.0 points on the reading component of the state- 
mandated 8th grade assessment. The knowledge gap between the special needs 
districts and the rest of the state on reading component of the 8th grade assessment is 
now 33.4 points. 

Writing. When NJSSI was birthed, the knowledge gap between the special need 
districts and the rest of the state on writing was measured at 53.1 points on the state 

mandated assessment. Over the next four years, there was a downward tend in the 

mean score on the writing section of the 8th grade assessment. In 1994, the mean 

score for students in special needs districts was: 114.0 in 1994; 121.1 in 1995; 113.6 in 

1996; and 109.9 in 1997. Over the same period, there was a similar downward trend in 
the mean score on the writing section on the 8th grade assessment for students 
enrolled in all other districts. The mean score for non-special needs districts was: 

167.1 in 1994; 176.2 in 1995; 160.6 in 1996; and 163.9 in 1997. During the years the SSI 

has been in operation, the knowledge gap between the special needs districts and the 

rest of the state increased by less than 1 point on the writing component of the state- 
mandated 8th grade assessment. 

Summary . In summary, during the time that NJSSI was in operation, the 
knowledge gap increased in mathematics, stayed about the same in writing (albeit 
with declining scores) and decreased in reading. 

Importantly, NJSSI management knew about the increasing knowledge gap between 
the special needs districts and the rest of the state on the mathematics component of 
the 8th grade assessment. The knowledge gap was documented year after year in the 
summaries of the state mandated 8th grade assessment in mathematics (March 1996 
Grade 8 Early Warning Test (EWT) State Summary, 1996; March 1997 Grade 8 Early 
Warning Test (EWT) State Summary, 1997). Despite seeing repeated documentation 
that the knowledge gap was increasing in mathematics, management of NJSSI 
ignored the problem. 



Science 



The New Jersey State Department of Education (NJDOE) does not currently assess 

science^. Science achievement will be assessed when the 4th grade assessment goes 
"live", in 1999. The NJDOE plans to add science to the 8th and 11th grade assessments 
in the near future. Overall, there is virtually no evidence concerning whether NJSSI 
is having an impact with respect to science achievement. Additionally there is no 
evidence whether the science impact is positive, neutral, or negative. 

NAEP recently released state breakdowns on science. Unfortunately, the 1997 data did 
not include New Jersey. Horizon Research Inc., NJSSIs external evaluator, conducted 

a small exercise, using unvalidated, non-standards based science questions for 4th 
and 5th grade students. Results from these exercises revealed that students of 
teachers that underwent SSI involvement scored significantly better than teachers 
who had volunteered to undergo SSI training, but had not yet received such training 
in 1996 (23 classes, 489 students), and significantly worse in 1997 (27 classes 485 

students)^. Realistically, nothing can be concluded from these exercises. 

Importantly, NJSSI's impact on science achievement is unknown. 

Management Issues: Demand for NJSSI Services 

Professional Development 

NJSSI had limited and decreasing capacity to deliver intensive professional 
development to the state of New Jersey. By 1998, the capacity of the seven 
institutions of higher education (IHEs) still participating in the Thrust I 

(professional development program) was only 240 seats^. However, demand for these 
seats was far lower than supply. While the final numbers of participating teachers 
for 1998 are not yet in, the cohort started in the spring of 1998 with only 70 teachers 
across all seven professional development sites combined. In 1998, NJSSI's 
professional development program was running at 29% of capacity. 

In the first four years, NJSSI paid full cost for teachers participating in the 
professional development program. Additionally, the NJSSI awarded participating 
teachers a "mini-grant" which enabled them to buy $500 worth of material for the 
classroom. Districts did not have to pay anything for training sponsored by NJSSI. 

As far as the districts were concerned, the NJSSI professional development program 

could be seen as a "free good"**. By paying full cost, the SSI allowed sites to get up and 
running quickly. However, some institutions of higher education become completely 
dependent on the SSI for the program, preventing the sustainability and 

institutionalization of the professional development initiative after federal and state 
funds run out. NJSSI tried to reduce its costs, and increase the sustainability and 
institutionalization of the program by making two management changes in the 
program between 1997 and 1998. 

In 1998 NJSSI reduced the funding for each site by 50% (from $67,000 to $33,000), 
expecting that the lost revenue to the professional development sites would be 

partially made up by requiring teachers to pay $500 each to receive the training. 

The $500 fee could be paid by teachers, or, more likely, school districts. Additional 

money was saved by eliminating the mini-grants. 
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During focus groups held in the summer of 1997, 50% of the participants warned 
NJSSI that their district would not have paid them to attend the training, and without 
district payment, they would not have participated. Similarly, the professional 
development sites were concerned that the program could not be sold to districts. 
Management of NJSSI ignored this information, and never asked school districts 
their interest in picking up part of the costs to fund the program. To date, the 
strategy of cost sharing between participants involved in professional development 
and the SSI can be evaluated as quite a bit less than successful. 

Across all seven sites, only 70 teachers have signed up to receive SSI training, about 
0.2% of the targeted market of science, math and technology (SMT) teachers 
throughout the state of New Jersey. The districts are not coming forward with the 
$500 fee, and teachers are less interested in participating due to the lack of mini- 
grants. Combing 1997 and 1998, NJSSI will train around 300-400 teachers, less than 
1% of the targeted market of 42,000 SMT teachers in New Jersey. With attrition and 
retirement of previously trained teachers, the market share of teachers that received 
NJSSI intensive professional development training in the state dropped in 1997, and 
will continue to drop through the end of 1998. 

Cost Per Participant of NJSSI's Professional Development Initiative 



Year 


Cost to 


Number of 


Cost Per 




NJSSI 


Participants 


Participant 


1997 


$745,000 


213 


$3540 


1998 


$256,000 


70* 


$3657* 



* Number of participants in 1998 subject to change. Cost estimates may change as 
well. 

An Request for Proposal to scale-up 

In response to a mandate from the NSF to "scale-up" and work with more than 31 
districts in the state, NJSSI sent out a Request for Proposals (RFP) in early 1997 for 
implementation of standards-based curriculum and assessment (called Thrust III) to 
the (then) functioning 594 school districts in the state. Additionally, the RFP was 
sent to private and charter schools Sixty-two districts and four private schools 
responded to the RFP, requesting $645,700 of SSI money to engage in a professional 
development program based on standards-based curriculum. The SSI had allocated 
$750,000 for the program, so the total amount of money requested was below the 
amount allocated. The SSI was in a position to fund all proposals. Again, the supply 
of SSI services exceeded demand. 

At time of this writing, seven districts dropped out of the Thrust III (standards 
implementation) initiative. Eliminating these districts brought the total amount of 
money spent on the program to $550,000, about 27% less thane amount allocated. 

Statewide Services 

The SSI had a small program to provide statewide services to the Thrust II (systemic 
reform) initiative. About $185,000 was allocated to the program, but the SSI received 
few requests to participate in the program. Less than $100,000 was requested 
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statewide. Again, demand for SSI services was very low, and the SSI could not award 
all of its money. 

Summary: Demand for NJSSI Services 

Overall, the demand for NJSSI services by school districts and teachers was below 
supply. The SSI has been unable to award all of its goods and services, and has 

received few requests for its 1998 professional development program. One solution 

to the problem of low demand would be a reduction in supply. If NJSSI is renewed by 
the NSF, the SSI’s budget will likely be reduced by 30%. The reduction in budget will 
reduce supply, and should help correct the current imbalance between supply and 

demand. If demand remains constant, the decrease in supply of SSI's services after 
1998 will bring this market into equilibrium. 

Sources of Data 

The 1997 New Jersey School Report Card (an 8MB file) is on the Web, and can be 
accessed with any standard Internet browser (http://www.evalsoft.com/NJ). The NJ 
School Report Card has a wealth of information on all 2,257 schools in the state. For 
middle schools, information on the 8th grade state mandated assessment is included. 

For high schools, information on the 11th grade state mandated assessment, SAT 
scores, AP enrollments and passing rates on AP exams, and self-report information 

on the transition to early adult life are reported, to mention some of the more 

pertinent variables. Attendance (faculty and student) are also on the school report 
card. 

State summaries of the 8th and 11th grade state mandated assessments provided a 
common base of tabular information that is used by NJDOE and state policy makers. 
Additionally, the state summaries provided information on the socio-economic status 
indicator used by the NJDOE for each school district in the state. The socio-economic 
status indicator was merged, at the school level, into the NJDOE school report card. 

Finally, programmatic information on NJSSI, including school districts sill involved 
with various SSI programs, school districts that dropped out of the SSI program, as 
well as number of teachers involved with the NJSSI's professional development 
program was merged, at the school level, into the NJDOE school report card. 

Data Analysis 

State Mandated Assessments: 8th Grade 

The NJDOE measures student performance on the 8th grade assessment using three 
proficiency levels: level I (the highest level, indicating students demonstrated clear 
competence in critical thinking skills measured by the assessment), level II (students 
demonstrated minimal competence based on the assessment), and level III (below the 
state minimum based on this performance). The low gain indicators track movement 
out of level III. The signs for the low gain indicator have been changed so that a 
positive score indicates a decreased percentage of students in level III and a negative 
score indicates an increased percentage of students in level III. We note here that 
the mathematics scores on the 8th grade assessment were the lowest of the three 
disciplines assessed. Scores on the writing assessment dropped, but 59% of the 
regular students in the state still scored at the highest level. Math scores increased 
modestly, but only 44% of the regular students in the state scored at the highest level. 



The dependent variables represent percentage change scores. The percentage of 
students scoring a particular category in 1995 was subtracted from the percentage of 
students scoring in the same category in 1997. The change score calculated by this 
process had a theoretical range from -100 to +100. However, to score -100, a school 
needed to move from 100% of the students from a particular category in 1995 (say. 
Math Level III) to 0% of the students in the same category in 1997 (say. Math Level 
III). To score +100, a school needed to move from 0% of the students in a given 
category in 1995 (say, Math Level III) to 100% of the students in a given category in 
1997 (say, Math Level III). No New Jersey schools showed anything close to these 
movements. Table 1 shows the mean change score for each component of the 8th 
grade assessment. 

A math low gain of 4.31 meant, on average, that 4.31% fewer students in a school 
scored at Level III, the lowest proficiency. A math high gain of 4.86 meant that an 
additional 4.86% of the students in a school scored at Level I, the highest proficiency. 
A negative score associated with a high gain meant that fewer students scored level I 
in 1997 than scored level I in 1995. A negative score associated with low gain meant 
that a greater number of students scored level III in 1997 than scored level III in 
1995. Table 1 presents a summary of the six change scores. 



Table 1: Changes in 8th Grade State Mandated Assessment Scores, 

1995-1997 



Subject 


Increase (+) or Decrease (-) in Score 


Math Low Gain 


4.31 


Math High Gain 


4.86 


Reading Low Gain 


0.98 


Reading High Gain 


-1.49 


Writing Low Gain 


-4.865 


Writing High Gain 


-8.922 



Table 2a: Regression Models Predicting to Increasing Scores on 

8th Mathematics Component of the State Mandated 

Assessment 





Dependent Variables | 


Independent Variables 


Math Low Gain 


Math High Gain 


Constant 


7 9 * * * 


5.735*** 


Thrust I 


-2.306 


-1.01 


Thrust II 


0.993 


-3.14 


Socio-economic Status 


- 1 1 9 * * * 


-.003 


R2 


.079 


.008 


N 


446 


446 | 
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Table 2b: Regression Models Predicting to Increasing Scores on 

8th Reading and Writing Component of the State Mandated 

Assessment 





Dependent Variables 


Independent 

Variables 


Reading Low 
Gain 


Reading High 
Gain 


Writing 
Low Gain 


Writing 
High Gain 




Constant 


2.357* 


.761 


-8.105*** 


-10.142 




Thrust I 


.633 


.049 


-.043 


1.743 




Thrust II 


.005 


-.333 


-.004 


1.415 




Socio-economic 

Status 


-.385 


358 


425 


425 




R2 


.018 


.002 


.050*** 


.001 




N 


358 


358 


425 


425 





Entries in the regression tables associated with the independent variables are 
unstandardized beta coefficients. 

* means p<.05 
** means p<.01 
*** means p<.001 

None of NJSSI's programs had a significant impact in improving educational 

performance in the state mandated assessments on mathematics, reading or writing. 
As shown in Tables 2a and 2b, not only were all 12 coefficients representing NJSSI's 
programmatic impact statistically insignificant, but six of them (50%) had the wrong 
sign! NJSSI's programs were as likely to have a negative impact on student 
achievement as a positive impact. Mathematics had the greatest number of negative 

coefficients. In mathematics, NJSSI's Thrust I (professional development) program 
had a NEGATIVE impact in improving educational performance in both the low and 
high gain indicators. NJSSI's Thrust II (systemic reform) program had a very small, 
statistically insignificant, positive coefficient on the low gain indicator, and a 
bigger, statistically insignificant NEGATIVE coefficient on the high gain indicator. 
The negative coefficients show that non-SSI schools made larger positive gains in 
student achievement than the SSI schools. 

Caveat: Extent of NJSSI’s Reach 

We have discussed NJSSI's reach as if such reach is a dichotomy; it happened, or did 
not happen. In reality, NJSSI reached schools very unevenly. In most cases, NJSSI 
had no reach into a school. In middle schools, NJSSI had some reach in 276 (46%) of 
593 schools. However, in 122 of those 276 schools (44%), the SSI's reach was limited to 
professional development experiences with 1-2 teachers. Such superficial reach may 
not be sufficient to positively alter a school's educational achievement or 
instructional program. A couple of teachers may be below the threshold necessary to 
create a critical mass of "change agents" (Berman and McLaughlin, 1974) required to 
effect educational change. 

The NSF understood this point, and required all SSIs to report the number of "focal 
schools". The focal schools, as understood by this author, was an attempt by the NSF 
to require SSIs to count schools they had reached at a depth greater than superficial. 
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We analyzed our data to account for levels of reach. We found no difference in 
student achievement when comparing Tables 2a and 2b to NJSSI's focal schools. 

However, we did find a greater impact in achievement when limiting our analysis to 
schools that had at least 7 teachers undergo NJSSI training. The student achievement 

effect was small, and the number of middle schools with 7 or more teachers trained 

by NJSSI limited (75 schools, or 12.6% of the total). However, the achievement effect 
was discernible. 

These results highlight a fundamental tension in SSI programs; depth vs. breadth of 
reach. Early in the program, the NSF wanted SSIs to focus on depth. Working 
intensively with a limited number of schools, the SSI would implement an 
educational reform initiative that eventually would positively impact a school. The 
change model would then be scaled-up in a number of ways: the SSI could branch out 
and work with other schools, or schools not involved in the program would see gains 
made by those involved in the program, and make changes to mirror the other 
school’s success. However, as the SSI program evolved, breath became more 
important than depth. The new strategy was to work with as many schools as 
possible, even if it meant that each school was only getting a very small benefit from 
affiliating with an SSI. 

In response to different NSF mandates over the years, NJSSI changed gears and 
focused on breath over depth. As a result, NJSSI spread itself too thin, and had 
superficial reach in 201 out of 276 middle schools (73%) where the SSI was at least 
minimally active. NJSSI did not positively impact student achievement in these 

schools. We note here the NSF's classification of focal schools was insufficient to 
impact student achievement. Five teachers, or 25% of a school's SMT teachers were 
not enough to effect educational change. Training of seven (or more) teachers was 
needed before a school began to show positive growth curves in student 
achievement, over and above what was expected by chance and socio-economic 
characteristics of students. 



NJSSI Impacts at the High School Level 



Enrollment in Advanced Placement Courses and Passing Rates on Advanced 
Placement Exams: Calculus 

Table 3 presents the analysis of Advanced Placement enrollments and passing rates 
on Advanced Placement exams in the state of New Jersey. Between 1995 and 1997, 
enrollments in AP calculus increased 8.3%, from 4,954 students to 5368 students, 
statewide. However, the passing rate on the AP exam dropped from 69% to 67%. The 
passing rate dropped across the entire socio-economic spectrum. The sharpest drop 
was observed in the urban districts (6%), and the smallest drop was observed in 
working class districts (1%). 

Urban schools affiliated with the SSI experienced decline in number of students 
taking the AP Calculus course (see figure 4) and a 7% decline in the passing rate 
(20% in 1995, 13% in 1997). Urban schools with no SSI affiliation experienced an 
increase in enrollment, and a 6% decline in the passing rate on the AP calculus exam 
(29% in 1995, 23% in 1997). 



O 

ERIC 



15 19 



Table 3: 



Advanced. Placement Enrollments, and Passing Rates on 
AP exams, State of New Jersey, 

1995-1997 



Subjects 


Enrollment 

1995 


Enrollment 

1997 


Passing 
Rate on AP 
Exam 1995 


Passing 
Rate on AP 
Exam 1997 


Calculus 


4954 


5368 


69% 


67% 


Biology 


3192 


4181 


69 


75 1 


Chemistry 


1696 


2214 


67 


65 i 


Physics 


1123 


1686 


65 


56 


Computer Science 


282 


694 


50 


46 


English Literature 


4960 


5859 


73 


73 


English Language 


1102 


1668 


63 


73 


All Others Combined 


9208 


12456 


63 


73 












Total of all AP 
Courses Combined, 
Entire State 


26517 


34126 


68 


68 



Figure 4: Declining enrollment of AP Calculus courses in NJSSI 

Affiliated Urban Schools 1995-1997 
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Enrollment in Advanced Placement Courses and Passing Rates on Advanced 
Placement Exams Biology 

Between 1995 and 1997, enrollments in AP biology increased 31.0%, from 3,192 

students to 4,181 students, statewide. The passing rate on the AP exam increased from 

69% to 75%. The passing rate increased across most of the socio-economic spectrum. 
The sharpest increase was observed in the most affluent suburban districts (9%). 

Students in urban schools increased their passing rate on the exam by 4%. 

Urban schools affiliated with the SSI experienced a 29% decline in number of 

students taking the AP Biology course, and a 3% decline in the passing rate (22% in 
1995, 19% in 1997). Urban schools with no SSI affiliation experienced a 22% increase 
in enrollment, and a 6% increase in the passing rate in AP Biology (23% in 1995, 29% 
in 1997). 

Enrollment in Advanced Placement Courses and Passing Rates on Advanced 
Placement Exams Chemistry 

Between 1995 and 1997, enrollments in AP chemistry increased 30.5%, from 1,696 

students to 2,214 students, statewide. The passing rate on the AP exam dropped from 

67% to 65%. The change in passing rate fluctuated across the socio-economic 
spectrum. The sharpest drop was observed in the most affluent suburban districts 
(8%). The passing rate for students in urban schools was unchanged between 1995 
and 1997 (27% in both years). Less affluent suburban districts experienced modest 
increases in the passing rates. 



Urban schools affiliated with the SSI showed a very small enrollment drop in AP 
Chemistry during 1995-1997, and a 1% increase in the passing rate on the exam (7% 
in 1995, 8% in 1997). Urban schools with no SSI affiliation experienced a small gain 
in enrollment, and a 1% decrease in the passing rate (37% in 1995, 36% in 1997). 

Enrollment in Advanced Placement Courses and Passing Rates on Advanced 
Placement Exams Physics 

Between 1995 and 1997, enrollments in AP Physics increased 50.1%, from 1,123 
students to 1,686 students, statewide. The passing rate on the AP exam dropped from 
65% to 56%. The passing rate increased most sharply among students in urban 
schools. In 1995, not one student from an urban school passed the AP Physics exam 
in the entire state of New Jersey! In 1997, 26% of students in urban schools passed 
the AP Physics exam. Decreases in the passing rate of 8% were observed in the 
working class and less affluent suburban areas. 



Urban schools affiliated with the SSI experienced a 200% increase in the number of 
students taking the AP Physics course, and an 11% increase in the passing rate (0% 
in 1995, 11% in 1997). Urban schools with no SSI affiliation experienced a 733% 
increase in enrollment, and a 37% increase in the passing rate (0% in 1995, 37% in 
1997). Figure 5 presents the trend lines. However, one should keep in mind that the 
analysis of AP physics enrollment, and passing rates on physics AP exams are based 
on small numbers. When looking for socio-economic differences these numbers 
become even smaller. The percentage increase in enrollments are impressive, but 
based on a slender empirical base. 



O 

ERJC 



17 



21 



Figure 5: Increasing Percentage of Urban Students Passing the 

Advanced Placement Physics Exam, 1995-1997 




YEAR 



Enrollment in Advanced Placement Courses and Passing Rates on Advanced 
Placement Exams Computer Science 

Between 1995 and 1997, enrollments in AP Computer Science increased 146.1%, from 
282 students to 694 students, statewide. The passing rate on the AP exam decreased 
from 50% to 46%. The sharpest drop in the passing rate on the Computer Science 
exam, 14%, was observed in the urban schools (25% in 1995, 11% in 1997). The 
sharpest increase in the passing rate on the Computer Science exam, 4% was 
observed in the affluent suburban schools (58% in 1995, 62% in 1997). 

However, one should keep in mind that the analysis of computer science AP 
enrollment, and passing rates on computer science AP exams were based on very 
small numbers. There was an insufficient number of urban students taking the AP 
Computer Science course and exam, precluding a comparison of SSI affiliated, and 
non-SSI schools. 

Enrollment in Advanced Placement Courses and Passing Rates on Advanced 
Placement Exams English Literature 

For comparison purposes, we include a discussion of AP enrollments of English 
Literature, the most popular AP course in New Jersey, and an analysis of all other AP 
courses combined. This comparison will enable the reader to assess AP enrollments 
and passing rates for subjects other than science, math or technology. 
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Between 1995 and 1997, enrollments in AP English Literature increased 18.1%, from 
4,960 students to 5,859 students, statewide. The passing rate on the AP exam remained 
stable at 73%. There was some fluctuation in the change in passing rate across the 

socio-economic spectrum. A drop of 3% was observed in the most affluent suburban 
schools (90% in 1995, 87% in 1997). The passing rate for students in urban schools 
was essentially unchanged between 1995 and 1997. 



Urban schools affiliated with the SSI showed an increase of 15.8% in AP English 
Literature enrollment, and a 1% drop in the passing rate on the exam (24% in 1995, 
23% in 1997). Urban schools with no SSI affiliation experienced a larger gain in 
enrollment (48.9%), with the same 1% decrease in the passing rate on the exam. 



Enrollment in Advanced Placement Courses and Passing Rates on Advanced 
Placement Exams All other AP courses combined 

Between 1995 and 1997, enrollments in all other AP courses increased 35.3%, from 
9,208 students to 12,456 students, statewide. The passing rate on the AP exam 
remained stable, at 66%. The change in passing rates for all other AP courses 
combined fluctuated narrowly across the socio-economic spectrum, with no major 
differences in any particular group. 

Urban schools affiliated with the SSI showed an 63.1% increase in enrollment for all 
other AP courses combined, and a 16% increase in the passing rate on all of the 
other AP exams combined (27% in 1995, 43% in 1997). Urban schools with no SSI 
affiliation experienced a 98.0% increase in enrollment, and a 6% drop in the passing 
rates on all of the other AP exams combined (42% in 1995, 36% in 1997). 



Table 4: Summary: Comparison of Performance Gains between 

Urban Schools Affiliated with NJSSI, and Urban Schools 
with no NJSSI Affiliation in Science, Mathematics and 
Technology (SMT) AP courses, 1995-1997 



Subject 


Schools 
Affiliated 
with NJSSI 


Schools 
with no 
NJSSI 
Affiliation 


No Difference 


Calculus 




Advantage 




Biology 




Advantage 




Chemistry 






Very Close 


Physics 




Advantage 




Computer Science 






Numbers too 
small for 
analysis 


TOTAL 


0 


3 


2 
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As summarized in Table 4, urban schools with no SSI affiliation showed greater gains 
in AP enrollments and sharper increases in passing rates on SMT AP exams than 
urban schools affiliated with NJSSI. In no case, did the urban schools affiliated with 
the SSI outperform urban schools without NJSSI affiliation. 



Table 5: Equity Issues in passing rates on AP exams. 



Subject 


Knowledge 
Gap in 1995 


Knowledge 
Gap in 1997 


Change in 
Knowledge Gap, 
1995-1997 


Calculus 


59% 


63% 


Increased by 4% 


Biology 


59% 


64% 


Increased by 5% 


Computer Science 


33% 


51% 


Increased by 18% 


Physics 


79% 


56% 


Decreased by 23% 


Chemistry 


58% 


50% 


Decreased by 8% 



Equity 



We computed a knowledge gap by comparing the passing rate on AP exams from 
schools in the most affluent suburbs with the urban schools. In all cases, the affluent 
suburbs had substantially higher passing rates than urban schools. We wanted to 
assess whether the knowledge gap increased, decreased, or stayed the same during 
the time NJSSI was in operation. 

The results of this analysis are shown in Table 5. In 1995, the knowledge gap was 
considerable in all subject matters, ranging from a low of 33% in Computer Science, 
to 79% in Physics. In 1997, the knowledge gap was at least 50% in all subjects. We 
note here that the knowledge gap increased in 3 out of the five subjects; Calculus, 
Biology and Computer Science. The knowledge gap decreased in 2 out of 5 subjects; 
Chemistry and Physics. Lastly, we soberly acknowledge the low passing rates of . 
students in urban schools in all SMT Advanced Placement exams. 

Student Performance on the High-Stakes 11th Grade State Mandated Assessment 

Table 6: Passing Rates on the High-Stakes, 11th Grade State 

Mandated Assessment, 1994-1996 



Year 


Math 


Reading 


Writing 


All 

Sections 

Combined 


1994 


84.4% 


83.2% 


88.5% 


73.2% 


1995 


86.2 


83.4 


90.4 


75.6 


1996 


85.9 


83.4 


90.5 


74.8 


Change in Passing 

Rates, 1994-1996 


1.5% 


0.2% 


2.0% 


1.6% 



Note: The 11th grade assessment is given in October and again in April. The October 

1997 and April 1998 results from this assessment will not be available until December 

1998 (or later). 
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The 11th grade state mandated assessment in New Jersey is a high stakes test for 
students— students have to pass the assessment to graduate. As reported in Table 6, 
nearly 3/4 of students pass all components of the assessment. Writing has the 
highest passing rate, reading the lowest. The passing rate of the mathematics 
component of the assessment dropped by a very small amount, 0.3% in 1997. It was 
the first time that the passing rate in mathematics dropped since the assessment was 
moved to the 11th grade. 

Klagholz noted (see Chiles, 1998) and we recognize, that the analysis of passing rates 
of the 11th grade assessment suffer from ceiling effects. Passing rates on the 
assessments, especially from schools in affluent suburban communities, exceed 95%. 
Schools in affluent suburban communities have little room for additional 
improvement. However, schools in special needs districts have plenty of room for 
improvement. Passing rates for the mathematics and reading sections are between 
55% and 60% for schools in special needs districts. The passing rate on the writing 
section is 74.0% in the special needs districts. 

Table 7: Regression Analysis of Changing Passing Rates on the 

High-Stakes, 11th Grade State Mandated Assessment, 

1994-1996 





Component 


Independent 

Variables: 


Math 


Reading 


Writing 


Constant 


.587 


-3.750 


-.783 


Thrust II 


1.845* 


.515 


.685 


Socio-economic Status 


.001 


.59 1 * * * 


.130 


Mobility 


.000 


.002 


.135*** 


N 


299 


299 


299 


R2 


.013 


.062*** 


.084*** 



Entries in the regression tables associated with the independent variables are 
unstandardized beta coefficients. 

* means p<.05 
** means pc.Ol 
*** means pc. 001 

With respect to mathematics, Table 7 shows that schools affiliated with the NJSSI had 
a 1.845% increase in the passing rate on the state mandated assessment, controlling 
for socio-economic status of a school and mobility. The 1.845% increase was (just 
barely) statistically significant. Neither mobility, nor socio-economic status 
explained changing passing rates on the math component of the 11th grade state 

mandated assessment. An of .013 indicates the model has virtually no 
explanatory ability to predict changing passing rates on the mathematics 11th grade 
state mandated assessment, and the entire equation is statistically insignificant. 
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With respect to reading, Table 7 shows that schools affiliated with the NJSSI had a 

0.515% increase in the passing rate on the state mandated assessment, controlling for 

socio-economic status of a school and mobility. The 0.515% increase was statistically 
insignificant. Mobility had no explanatory power to explain changing passing rates 
on the state mandated assessment. Socio-economic status did have a significant 

impact on changing passing rates. Every step up the socio-economic ladder resulted 
in a 0.591% increase in students passing the reading assessment, controlling for 

mobility and the SSI program. An R 2 of .062 indicates the model has virtually no 
explanatory ability to predict changing passing rates on the reading 11th grade state 

mandated assessment, even though the equation, taken as a whole, is statistically 
significant. 

With respect to writing, Table 7 shows that schools affiliated with the NJSSI had a 

0.685% increase in the passing rate on the state mandated assessment, controlling for 

socio-economic status of a school and mobility. The 0.685% increase was statistically 
insignificant. Socio-economic location of a school had no explanatory power to 
explain changing passing rates on the state mandated assessment. Mobility did have 
a impact on changing passing rates, but in the opposite direction to what was 
hypothesized. Higher rates of student mobility were associated with an increased 

passing percentage. An R 2 of .084 indicates the model has virtually no ability to 
predict changing passing rates on the reading 11th grade state mandated assessment, 

even though the equation, taken as a whole, is statistically significant. 

In summary, we see a very small, positive, but discernible and (just barely) 
statistically significant effect for NJSSI's Thrust II program in increasing passing 
rates on the 11th grade state mandated assessment in mathematics. We see some very 
small, and statistically insignificant effects for NJSSI's Thrust II program in reading 
and writing. We note here that the coefficients representing NJSSI's programmatic 
impact in predicting increased passing rates in reading and writing were positive. 

These positive coefficients could be interpreted as very small spillover effects from 
the more narrowly focus disciplines the SSI tried to impact positively— math and 
science. Lastly, it could be argued that we are estimating population values (the 
NJDOE includes information on all 383 high schools in the state) and statistical tests, 
and statistical inference have no meaning for this data set. 

Scholastic Aptitude Test Results 

The mean score in New Jersey for the mathematics component was of the SAT exam 
was unchanged between 1995-1997. In 1995, the mean score was 511. The mean score 
dropped 3 points in 1996, and increased 3 points in 1997. The percentage of high 
school seniors taking the SAT in New Jersey was: 73% in 1995, 75% in 1996 and 74% in 
1997. With respect to math, we see a 1% increase in the percentage taking the SAT, 
and no change in mean score. 

Models predicting to changing SAT scores were estimated. The outcome of those 
models were similar to the 11th grade state mandated assessment. The SSI's program 
was unrelated to changing SAT scores in every model. No spillover effects were 
detected—with respect to the verbal component of the SAT, we found that schools 
affiliated with NJSSI had consistently lower gains on the verbal SAT, but those effects 
were small, and statistically insignificant. All estimated coefficients were 
statistically insignificant. 
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Transition to Early Adult Life 



From the launch of Sputnik to the present day, there has been the argument that the 
United States needs to be first in the world in achievement in math, science and 
technology. It has been postulated that i_J_ the U.S. was not first in the world in SMT 
achievement, that the economic vitality of the country would somehow be put at risk. 
The argument follows the following lines. We all have a vested interest in improving 
SMT education. Failure to improve SMT education risks a downturn in our standard of 
living sometime in the next millennium. An evaluation of these statements are 

beyond the scope of this paper, but it is important to note that programs to improve 

SMT education continually get strong support from the U.S. Congress, the business 
community, and the general public due to these arguments. 

Impact in state mandated assessments, advanced placement classes, advanced 
placement exams, and SAT scores are all specific and narrowly focused academic 

indicators to assess the impact of an SSI's short term instructional program. Longer 

term, SSIs would like to claim that its benefits are broader than particular courses 
taken, or assessments passed. SSIs would like to think that their benefits "stick to" 

students throughout their life cycle. Ideally, SSIs hope to facilitate improved life 

cycle outcomes for school children, and high school graduates though a better/easier 

transition to adult life (school to work, and secondary to post-secondary, to mention 
the two most common transitions to early adult life). From this broader perspective, 
state mandated test scores, AP exams, and SAT scores are but three indicators in the 
overall picture of student achievement. Additionally important indicators of the 
effectiveness of an SSI would be workplace readiness, graduation rates, percentage 

of students going on to college, and percentage of students with well defined career 
goals, to mention only a few examples from a very long list. 

Thanks to the work of the NJDOE, we can look at career goals through a limited 

measure of transition to early adult life. The NJDOE collects self-report data on high 
school graduates as they prepare to enter adult life. We frankly acknowledge the 
limitations of using self-report data, especially when prospective high school 

graduates may be likely to report that they will be going on to college (2 or 4 year) 
when post-secondary education is not currently in their plans. Despite this 

limitation, a focus on transition to early adult life allows analysts to look for 
programmatic effects of an SSI that are far broader than immediate school 
achievement. 

Table 8: Regression Analysis Predicting to Successful Transition 

to Early Adult Life 



Independent Variables 


b 


Std. Error 


Significance 


Constant 


81.607 


1.458 


p<.001 


Socio-Economic Status 


2.104 


0.263 


p<.001 


Mobility 


-0.080 


0.030 


p<.01 


Thrust II 


-1.662 


1.820 


Not 

Significant 



R 2 =.281 

N=290 
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Table 8 presents estimates from a regression equation predicting to successful 
transition to early adult life. Most students anticipate a successful transition to early 

adult life (moving from secondary school to post-secondary [2 or 4 year], or moving 

from secondary school to work). Factoring out the effects of socio-economic status, 
mobility and the NJSSI's Thrust II program, we predict the percentage of students in 
a school anticipating successful transition to adult life to be 81.6%. Each move up the 
socio-economic scale increases the percentage of students anticipating a successful 
transition to adult life by 2.1%. In affluent suburban schools (four moves up the 

socio-economic scale from urban), we predict 90% of the students anticipate a 
successful transition to early adult life. Mobility negatively impacts transition to 
early adult life. A 1% percentage increase in mobility decreases the estimated 
percentage of students anticipating successful transition to early adult life by .08. 
Finally, the results show that the NJSSI had no significant impact on transition to 
early adult life. High schools involved in NJSSI’s Thrust II program were predicted to 
have 1.6% fewer students anticipating successful transition to early adult life than 
schools with no SSI affiliation. 

We find no evidence that NJSSI successfully impacted the transition to early adult 
life. However, we recommend that analysts looking for programmatic impact of 
reform initiatives like SSIs continue to look for and investigate broad dimensions of 
possible effects. Exclusively focusing on limited, and more immediate outcomes (in 
an SSI’s case, performance on state mandated assessments, AP exams, AP passing 
rates, SAT scores, attendance, etc., does not capture some components of what an SSI 
was trying to accomplish.) The longer term impacts; workplace readiness, 
employability in the work force, confidence in new situation, if they can be shown to 
exist, would give reform initiatives far greater legitimacy than a demonstration of 
gains on a state-mandated assessment. 

Conclusions 



Conclusions Pertinent to NJSSI 

(1) NJSSI never came to an understanding on how to deliver intensive professional 
development to New Jersey's teachers. The SSI's 1994-1997 strategy— pay for 
everything and hope for the best was, at the same time E & L (expensive and limited). 
Paying $3500 per teacher for professional development drained the NJSSI budget 
quickly, and helped limit the reach of the initiative. The 1998 strategy— have school 
districts pick up some of the costs— did not work due to poor relations between the 
NJSSI and school districts. 

(2) Nobody employed at NJSSI had any subject matter expertise in urban education. 
Additionally, 88% of the people employed at the SSI lived in upscale, relatively 
wealthy suburban areas. Only one employee, and a relatively low ranking one at 
that, lived in one of special needs districts. From professional training, nobody 
employed in the NJSSI was qualified to deal with urban education. From personal 
experience, nobody employed at the SSI in a decision or policy making capacity had 
an understanding of the’ cultural, racial and economic factors that exert influence 
on special needs districts. These personnel issues may explain why NJSSI was not 
able to deal with the increasing knowledge gap found between the special needs 
districts and the rest of the state on the mathematics component of the 8th grade state 
mandated assessment. 





(3) The NJSSI attempt to positively impact 8th grade mathematics achievement was 
unsuccessful. Not only were all 12 estimated coefficients for SSI programmatic 
activity statistically insignificant, but 50% of them had the wrong sign! The 
statistical effects of the program were worse than what would be expected by chance 
in mathematics, and better than chance for writing. Overall, the statistical effect was 
equal to chance. 

NJSSI did had a small positive impact in improving 8th grade mathematics in schools 
where 7 or more teachers underwent the initiative’s professional development 
experience. Where a critical mass of trained teachers existed, a small positive impact 

for student achievement was detected. For NJSSI, the finding had no practical value. 
The cost of seven teachers undergoing the initiative’s professional development 
experience was $24,500 per school, and there are 2,257 schools in the state. The total 
cost to scale-up training in this manner, $55 million dollars exceeded by fivefold, 
NJSSI's total projected budget in years 6-10. NJSSI would have to get its costs under 
control before trying to use this recommendation to M scale-up". 

(4) There is no evidence concerning the NJSSI’s impact on science achievement in 
the state at any grade level. We don’t know if NJSSI's impact on science was positive, 
neutral, or negative. 

(5) The increase in students taking AP courses is one of the most positive 

components of educational change in New Jersey^. Impressively, there was no state 
policy to facilitate growth of AP enrollments, and NJSSI did almost nothing to 
increase student participation in any AP course. The increase in AP courses was due 

to collaborative efforts at the local level of the following actors: students requesting 

AP courses, parental support for the courses, a desire on the part of administrators to 
offer the courses, and finally, a desire on the part of the faculty to teach AP courses. 
This local collaboration has been most successful in the economically advantaged 
suburban areas. High schools in the most affluent communities in New Jersey often 
offer 10 different AP courses. This local collaboration has been least successful in 
urban areas, where schools often offer 4 or less AP courses. However, gains in AP 
enrollment span the entire socio-economic spectrum. 

The drop of 12.3% in AP calculus enrollments for urban schools affiliated with NJSSI 
is surprising, since the state exhibited strong enrollment gains for this subject from 
1995-1997, and urban schools with no NJSSI affiliation also showed gains in 
enrollment. It is possible that the negative AP enrollment trend lines observed in 

SSI affiliated urban schools could be due to policies advocated by the NSF and NJSSI. 
During the early years of the SSI program, NJSSI and NSF advocated school districts 
eliminate general math and science courses. Some school districts in New Jersey 

followed those recommendations. A couple of years later, these school districts 
determined that eliminating general math and science courses did not work, and 

reinstated these courses in the curriculum. However, it will take at least two years, 
and probably at least three years, to determine if reinstating general math and 
science courses positively impact course enrollments in Advanced Placement: 
Calculus, Biology, Physics, Chemistry and Computer Science. 

(6) About 11% of the school districts engaged in SSI programmatic activity dropped 
out of NJSSI in 1997-1998. One of those districts dropping out, Paterson, left the 
Thrust II program. As mentioned earlier, 85% of Paterson's students are eligible for 
free and reduced lunch, making this district one of the neediest in the state. The 
other districts dropping out of NJSSI left the Thrust III program. 





With 8 districts dropping of NJSSI, analysts using the NJDOE School Report Card in 
1998 can set up stronger research designs than those used in this paper to estimate 
the initiative's programmatic impact. For example, analysts could set up a removed 

treatment design with pre-test and post-test, the strongest non-experimental design, 
(see Cook and Campbell 1979) to ascertain if Paterson's growth curves on state 
mandated assessments were any different than the growth curves of urban schools 

remaining in NJSSI's Thrust II initiative. With 35 schools, Paterson, the third largest 

school district in New Jersey, has sufficient numbers of schools and students to 
permit analysts to successfully estimate these relationships. Analysts could also set 

up a three group design to estimate the impact of Thrust III; districts completing the 
program, districts dropping out of the program, and districts not involved in the 
program. We note here that the first analysis concerning the student achievement 

impact of Thrust III will be available when the 1998 data is released. These stronger 
research designs, available only because school districts dropped out of NJSSI 
programs, will enable the NSF to get a clearer picture of the impact of the initiative’s 
accomplishments. 

(7) SSIs could be detrimental to a school's educational improvement plan. Some 

schools are involved an abundance of programs to improve educational achievement. 

Additional educational improvement programs (SSI or non-SSI) can create 

management overload, a situation where there is not enough time or staff to 

implement any one program well. Teachers, especially in self-contained 

classrooms, need to keep these programs in their minds, sometimes all at once. 

Central office staff need to administer these programs. The problems of information, 
management and program overload is especially true in large urban districts. Large 
urban districts are involved in many programs, perhaps more than they can 

comfortably handle. Reducing the load of programs may be beneficial, especially for 

a school district like Paterson. From this perspective, tracking growth on state 
mandated assessments for districts/schools dropping out of NJSSI will be especially 
illuminating. 

(8) No voluntary educational reform effort can succeed without strong cooperation 
from the school districts. As mentioned before, the NJSSI’s relationship with school 
districts was very week. Demand for NJSSI services was very low. Low demand for 
services does not bode well for the future of NJSSI. 

(9) The SSI had no positive impact on equity. When the SSI was birthed in 1993, New 

Jersey had a substantial knowledge gap between performance of underserved and 
better served students. The statistical effect called "regression towards the mean" 
would predict that urban students would show some modest improvement, and 
affluent students from suburban schools would show some modest decreases in 

performance, even if NJSSFs equity programs were unsuccessful. Incredibly, we 
find the knowledge gaps increasing. The knowledge gap increased with respect to 
performance on the state mandated 8th grade mathematics assessment. The 
knowledge gap also increased in passing rates in Advanced Placement (AP) Calculus, 
AP Biology, and AP Computer Science. With respect to equity, New Jersey is in worse 
shape today (in 1998) than 1993. 

Conclusions Pertinent to the NSF SSI program 

(1) NSF changed over the years on the question of whether the SSIs should focus on 
depth vs. breadth of activity at the school level. In the early years, NSF wanted depth 
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of activity. The SSIs were encouraged to create models for change, determine if those 
change models worked, and then scale-up successful components of the reform 

initiative. In more recent years, desiring to report big numbers of schools reached 

to the U.S. Congress (and other audiences), NSF stressed breadth of activity. SSIs were 
encouraged to scale-up activity to a wide number of schools. As a result of NSF's 
change in strategy, SSIs may have spread themselves too thin. Our analysis suggests 
that reaching one or two teachers is not sufficient to impact student achievement in 
a school. We estimate that seven teachers are needed before a small positive impact 
on achievement exists. If the case of NJSSI can be generalized to other SSIs, many 
schools across the country have been superficially impacted by the SSI program due 
to the strategy of NSF stressing breadth over depth. Therefore, we recommend that 
the NSF revisit their managerial strategy, stressing breath over depth. 

(2) It might be important to note that the NSF's classification of focal schools, (the 

training of 5 teachers [or 25% of the science or math teachers] in a school), was 
insufficient to positively impact a school's scores on state mandated assessments. If 

the focal schools indicator was an attempt to assess depth of an SSI's activity, we hope 
the NSF would consider investigating what kind of training, and with what numbers 
of teachers, would be required before a "critical mass" of "change agents" exist in a 
school. Clearly, the NSF would not want to set the minimum number of trained 
teachers required to classify a school as focal too high. Setting the number for focal 
schools too high would create low reach numbers for SSIs, and create political 
problems for NSF when dealing with other audiences, most notably the U. S. Congress. 
On the other hand, setting the reach numbers too low sets up a situation where the 
NSF is claiming that SSIs intensively reached a school, yet indicators on student 
achievement indicate that no positive accomplishment resulted from SSI activity. 

(3) The SSIs are limited by NSF's focus on science and mathematics. Support of the 
school principals— often noted as a critical determinant of successful reform 
initiatives— was lacking. NJSSI hoped, and requested, that principals attend the 
summer training institutes. In visits to the training sites, only 3 principals were 
present, out of 213 participants. Principals will not spend a substantial amount of 
their time in programs exclusively limited in disciplinary focus. Principals take a 
broader disciplinary perspective, and invest their time in programs that are wide 
ranging. 

(4) The NSF wanted a number of different actors to work together towards a common 
goal of improving SMT education. Curiously, the NSF did not practice what it 
preached when implementing the program. The NSF did not work with another 
major government agency with a strong interest, and a history of attempting to 
improving k-12 educational outcomes, the US Department of Education (USDOE). An 
affiliation between the USDOE and NSF could have given the SSI program a broader 
disciplinary focus, and legitimate educational improvement efforts in reading, 
writing, and English. 

We note here the NSF and USDOE have very recently announced that they will work 
together "to implement an action strategy to support local efforts to put in place the 
rigorous courses and effective teaching that will build a strong foundation in middle 
years" (http://www.ed.gov/TIMSS/overview.html). In the future, we hope that these 
collaborative efforts will yield dividends. Additionally, we hope that analysts of the 
USDOE-NSF joint program will not have to go out of their way to look at "spillover" 
effects— effects outside the prime domain of NSF— in order to analyze improved 
student performance in reading and writing. With the help of the USDOE, we hope 
that future NSF funded educational reform initiatives will embed the strong 
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expectation that all disciplines: science, math, technology, reading, English, writing, 
and others, will all show improvements. 

(5) Lack of accountability has been a major problem with the NSF SSI program. 

States like New Jersey claim to have reached 80%+ of the teachers and school 
children in the state, but the NSF has no mechanism to easily verify such claims. Our 
analysis showed that NJSSI reached, at best, about 20% of the school teachers in New 
Jersey. In a competitive situation, where the NSF will only renew a limited number of 
SSI's, an accountability mechanism to verify state claims is essential. 

An even more important question than the reach of an SSI is whether the 

programmatic activity did any good. Did the SSI add value to a state's educational 
program? Our analysis shows that NJSSI's 8th grade math program did not impact the 
state positively. With data sets like the NJDOE school report card, the NSF could 
conduct comparative analysis on achievement gains, provide some accountability, 
and a greater sense of legitimacy in the student achievement area than simply 
accepting claims made by individual SSIs. 

(6) The public does not view science as the "basics". In one statewide random sample, 

88% of respondents responded that math would be classified as the basics. Science 
was classified as the "basics" with 43% of respondents (1995 Colorado Education 
Standards Survey). States assess the basics, and, occasionally, subjects outside the 

basics. Not every subject can be assessed. New Jersey only assess the basics; reading, 

writing and mathematics. Science is not yet assessed. Science competes with a 
number of non-basic disciplines for the honor to be assessed. The decision to assess 
science creates a burden and cost for a state. If the responses of the Colorado 
Education Standards Survey are typical, states do not face much public pressure to 
assess science. Science does not have the same stature with the public as 
mathematics. It will take the NSF, and the rest of the federal government, quite a 
number of years to heighten public awareness on the importance of assessing 
science. 

(7) What are the limitations of SSIs? With a $3 million yearly budget, NJSSI tried to 
influence k-12 education in New Jersey, a $12-13 billion dollar a year business. Put 
another way, for every $4000 spent on k-12 public school education in New Jersey, 
NJSSI spends $1. What is realistic to expect for that $1? At one level, the NSF got a 
few paid lobbyists in a state, attempting to influence the political and bureaucratic 
machinery of government to alter SMT education in New Jersey towards NSF's view 
(active learning). Beyond the political dimension, could NJSSI realistically be 
expected to make an impact on equity or student achievement with a $3 million dollar 
budget? We found that NJSSI made virtually no positive impacts on either of these two 
critical issues. NJSSI clearly fell short on these dimensions, with limited reach, weak 
programs, and an insufficient diversity of people knowledgeable on issues 
pertaining to urban education. 

Concluding Summary: The CPRE Questions 

CPRE raised three evaluation questions pertinent to the SSIs. 

1. Are students learning more because of these SSIs? 

2. Are students better equipped to apply what they are leaning to everyday 

problems in the community and in the workplace? 
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3. Are the inequities in performance among different groups of students being 

reduced? 

In summary, we find no evidence for question number 1. Students in schools 
affiliated with NJSSI did not learn more than their counterparts enrolled in schools 
with no NJSSI affiliation. There was no value added in the NJSSI schools. We have 
only very limited evidence with respect to evaluation question number 2. The 
evidence comes from the information on the transition to early adult life, where we 
found that students in SSI affiliated schools were slightly less likely to expect a 
successful transition to early adult life than students in schools not affiliated with 

NJSSI. We would feel more comfortable concluding that we don't know the answer to 
question 2. Question 3 was easily answered by the data. Inequities in performance 

got worse. The equity situation in New Jersey is worse today than in 1993. 
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Endnotes 



1: Given the history of educational initiatives, the renewed SSIs should expect, and 

anticipate, another shift from the NSF about 4 years. Educational fads are in vogue 
for about five years. Renewed SSIs from Cohorts II and III may have to face that fact 

that the standards-based educational reform movement will not live up to 

expectations, and the NSF may embrace yet another fad around the year 2002. 

2 The court ruled that two special needs districts, Hoboken and New Brunswick were 
adequately funded. These two districts did not receive any additional funds from the 
court ruling. 

3. We note here that the management of NJSSI uses a different counting system. 

NJSSI will claim impact for an entire district (100% of teachers and 100% of students) 
with as few as one teacher going through a professional development experience. 

For example, in Toms River, one teacher went through a New Teacher Institute 
program. [We will not deal with the question of whether this non-SSI training 
should count in the SSI professional development.] Due to the activity of one teacher, 
NJSSI will claim to have impacted the entire district of Toms River, a district with 
17,000+ students and 1,000+ teachers! NJSSI never postulated an innovation or 
diffusion model that would explain how one teacher could have had such an 
impressive impact on a school district. It was simply asserted that if one teacher goes 
through an experience, the entire district has been reached. It is through this 
counting procedure— reaching one teacher in a district means the SSI can claim 
100% of the students and teachers— that the SSI claims to reach over 80% of the 
students and teachers in New Jersey. 

When it comes to estimating commitment to standards-based curriculum, the NJSSI 
has an even bigger leap of faith. The NJSSI will claim that a district exhibits a strong 
commitment towards standards-based curriculum and instruction with as few as one 
teacher going through Thrust I. NJSSI will make that claim despite the fact that the 
professional development training (called Thrust I) had no standards-based 
instruction in the first three years, and required no district commitment of 

curriculum change. Over 98% of the intensive professional development at NJSSI 
occurred when the initiative was striving for systemic reform. Only 2% of the 
intensive professional development at NJSSI occurred when the initiative was 
considered standards-based. Overall, these issues bring up a fundamental problem in 

NSF's SSI program, the lack of accountability. Particular SSI's can make claims, but 
the NSF has no way to verify the claims. We will deal with the accountability issue in 
the concluding section of the paper. 

4. Special needs districts were defined by the NJDOE through a principal components 
factor analysis of the following seven variables: percent of adult residents who failed 
to complete high school, percent of adult residents who attended college, 
occupational status of adult household members, population density, income, 
unemployment and poverty. A single measure of socio-economic status was then 
created from the factor analysis. Special needs districts come from the bottom of that 
socio-economic scale. 

5. NJDOE did include science in the pilot of the 4th grade assessment. Scores from this 

assessment were not released to the districts, nor to NJSSI. 
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6. One can easily criticize these research exercises in a number of important ways 

Subjects were not randomly selected, the control group was self-selected, so strictly 
speaking, statistical inference would be inappropriate. Additionally, there was no 

control in the way teachers implemented these research exercises. Thirdly, since the 
exercise was completed in one class period (typically 40-50 minutes), the domain of 
questions was limited. In some cases, only one question was asked. Fourth, all 
questions required a written answer, so there was no variation in the modality of 
responses. Lastly, the questions would not permit the SSI to claim that students were 
learning more. Based on the Horizon questions, NJSSI might have been able to claim 
that students in classes taught by NJSSI trained teachers were able to perform better 
on questions requiring a specific set of skills (e.g., performance based), than 
students in classes taught by teachers without NJSSI training. However, the Horizon 
exercises could not determine if gains in performance-based skills were offset by 
decreased skills in answering questions from any other type of assessment modalities 
(e.g., multiple choice, constructed open-response, essays, portfolios, and giving a 
speech). We would need to know if gains made on one modality of assessment were 
not offset by losses on other modalities of assessment before we could conclude that 
students learned more. The Horizon questions could only assess whether student 
achievement was different in classes taught by NJSSI trained teachers. Lacking the 
critical check of other assessment modalities. Horizon could not determine if gains 
made with performance-based work came at the expense of other type of knowledge. 
For these reasons, the Horizon questions were too limited to address the CPRE 
evaluation question concerning whether students learned more due to the SSIs. 

7. Some institutions of higher education voluntary dropped out of Thrust I, others 
were determined to perform at less than a satisfactory level and were phased out. 

8. Some districts did pay teachers a stipend for participating. 

9. Gains in Advanced Placement course enrollments are not limited to New Jersey. 
These gains have been experienced by many states, and represent a national 
phenomenon . 
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